NumPy is a Python library, which is mainly used for scientific computing. It contains a collection of tools and techniques that can be used to resolve number of problems in scientific computing.
It also provides a high-performance multidimensional array object, and tools for working with these arrays.
As its not a core python library, it needs to be installed. It can be installed using pip utility
$:> pip install numpy --user
To use Numpy, we first need to import the numpy package:
In [2]:
import numpy as np
One of tools provided by NumPy is ndarray. It is high-performance multidimensional array object that is a powerful data structure for efficient computation of arrays and matrices. To work with these arrays, there’s a huge amount of high-level mathematical functions operate on these matrices and arrays.
In other words, numpy array is a n dimentional grid of same data type values and is indexed by a tuple of non-negative integers.
In [144]:
a = np.array([1, 4, 5, 66, 77, 334], float)
print(type(a), a.shape)
print( a)
a[0] = 125 + 1/4
print( a)
In [44]:
am = np.array([[3,5,7,9], [2, 4, 5, 6]], float)
print(type(am), am.shape)
print(am)
In [9]:
am3= np.array([[3,5,7,9], [2, 4, 5, 6], [2, 6, 10, 14]], float)
print(type(am3), am3.shape)
print(am3)
In [12]:
am3= np.array([[3,5,7,9], [2, 4, 5, 6], [2, 6, 10, 23]], int)
print(type(am3), am3.shape)
print(am3)
numpy arrays can be initialize using nested Python lists, and elements can be accessed using square brackets, similar to python nested lists.
In [19]:
import numpy as np
a = np.array(range(2,20,4)) # Create a rank 1 array
print(type(a)) # Prints "<class 'numpy.ndarray'>"
print(a.shape)
for i in a:
print(i, end=", ")
print()
b = np.array([[2, 4, 6],
[11, 13, 15]]) # Create a rank 2 array
print(b.shape)
print(b[0, 0], b[0, 1], b[1, 0])
c = np.array([[[2, 4], [6, 8]],
[[11, 13], [15, 17]],
[[111, 113], [115, 117]]]) # Create a rank 3 array
print(c.shape)
In [21]:
b = np.array([[2, [4], 6],
[11, [13], 15]])
above failed as all the elements were not of same data type
Numpy also provides many other functions to create arrays
In [23]:
a = np.zeros((2,4)) # Create an array of all zeros
print(a)
print("*"*30)
b = np.ones((1, 4 , 2)) # Create an array of all ones
print(b)
print("*"*30)
c = np.full((2,5), 101) # Create a constant array
print(c)
print("*"*30)
d = np.eye(5) # Create an identity matrix
print(d)
print("*"*30)
e = np.random.random((1, 4)) # Create an array filled with random values
print(e)
print("*"*30)
e = np.random.random((2, 4)) # Create an array filled with random values
print(e)
Numpy offers several ways to index into arrays.
In [29]:
b = np.array([[2, 4, 6], [11, 13, 15], [112, 114, 116], [222, 444, 666]]) # Create a rank 2 array
print(b.shape)
b2 = b[1 : 2]
print(b2)
b3 = b[1 : 3 ]
print(b3)
b21 = b[1 : 2][0][0]
print(b21)
the last example does not look that convinent, so lets try the following one
In [30]:
print(b[1:2, 0])
print(b[1:2, :1])
print(b[1:2, ::2])
print(b[1:2, :])
In [72]:
ar = np.random.random((4, 2, 3))
print(ar)
print(ar.shape)
In [73]:
t1 = ar[1: 2]
print(t1)
print(t1.shape)
In [42]:
a = np.array([[1, 2], [3, 4], [5, 6]])
print(a)
print("*"*20)
# print(a.shape)
# print("*"*20)
# An example of integer array indexing.
# The returned array will have shape (3,) and
print(a[[0, 1, 2], [0, 1, 0]]) # Prints "[1 4 5]"
print("*"*20)
print(a[0, 0])
print(a[1, 1])
print(a[2, 0])
print("*"*20)
# The above example of integer array indexing is equivalent to this:
print(np.array([a[0, 0], a[1, 1], a[2, 0]])) # Prints "[1 4 5]"
print("*"*20)
# When using integer array indexing, you can reuse the same
# element from the source array:
print(a[[0, 0], [1, 1]]) # Prints "[2 2]"
print("*"*20)
# Equivalent to the previous integer array indexing example
print(np.array([a[0, 1], a[0, 1]])) # Prints "[2 2]"
One useful trick with integer array indexing is selecting or mutating one element from each row of a matrix:
In [43]:
import numpy as np
# Create a new array from which we will select elements
a = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])
print(a)
# Create an array of indices
b = np.array([0, 2, 0, 1])
# Select one element from each row of a using the indices in b
print(a[np.arange(4), b])
# Mutate one element from each row of a using the indices in b
a[np.arange(4), b] += 10
print(a)
In [79]:
import numpy as np
a = np.array([[1,2], [3, 4], [5, 6]])
bool_idx = (a > 2) # Find the elements of a that are bigger than 2;
# this returns a numpy array of Booleans of the same
# shape as a, where each slot of bool_idx tells
# whether that element of a is > 2.
print(bool_idx)
# We use boolean array indexing to construct a rank 1 array
# consisting of the elements of a corresponding to the True values
# of bool_idx
print(a[bool_idx])
# We can do all of the above in a single concise statement:
print(a[a > 2])
In [60]:
am = np.array([[3,5,7,9], [2, 4, 5, 6.101]])
bool_idx2 = (am > 5)
print(bool_idx2)
print(am[bool_idx2])
In [56]:
am = np.array([[3,5,7,9], [2, 4, 5, 6]], int)
bool_idx2 = (am >= 5)
print(bool_idx2)
print(am[bool_idx2])
In [55]:
print(am[am >= 5])
Every numpy array is a grid of elements of the same type. Numpy provides a large set of numeric datatypes that you can use to construct arrays. Numpy tries to guess a datatype when you create an array, but functions that construct arrays usually also include an optional argument to explicitly specify the datatype. Here is an example:
In [76]:
import numpy as np
x = np.array([1, 2]) # Let numpy choose the datatype
print(x.dtype)
x = np.array([1.0, 2.0]) # Let numpy choose the datatype
print(x.dtype)
x = np.array([1, 2], dtype=np.int64) # Force a particular datatype
print(x.dtype)
A data type object represent, fixed block of memory corresponding to an array, depending on the following aspects:
Numpy supports following data types, some of them might not be present in python.
| Data Types | Description | Range | |
|---|---|---|---|
| bool_ | Boolean (True or False) stored as a byte | 0 1 | |
| int_ | Default integer type (same as C long; normally either int64 or int32) | ||
| intc | Identical to C int (normally int32 or int64) | ||
| intp | Integer used for indexing (same as C ssize_t; normally either int32 or int64) | ||
| int8 | Byte | (-128 to 127) | |
| int16 | Integer | (-32768 to 32767) | |
| int32 | Integer | (-2147483648 to 2147483647) | |
| int64 | Integer | (-9223372036854775808 to 9223372036854775807) | |
| uint8 | Unsigned integer | (0 to 255) | |
| uint16 | Unsigned integer | (0 to 65535) | |
| uint32 | Unsigned integer | (0 to 4294967295) | |
| uint64 | Unsigned integer | (0 to 18446744073709551615) | |
| float_ | Shorthand for float64 | ||
| float16 | Half precision float: sign bit, 5 bits exponent, 10 bits mantissa | ||
| float32 | Single precision float: sign bit, 8 bits exponent, 23 bits mantissa | ||
| float64 | Double precision float: sign bit, 11 bits exponent, 52 bits mantissa | ||
| complex_ | Shorthand for complex128 | ||
| complex64 | Complex number, represented by two 32-bit floats (real and imaginary components) | ||
| complex128 | Complex number, represented by two 64-bit floats (real and imaginary components) |
built-in data type has a character code that uniquely identifies it.
| Char code | Data Types |
|---|---|
| 'b' | boolean |
| 'i' | (signed) integer |
| 'u' | unsigned integer |
| 'f' | floating-point |
| 'c' | complex-floating point |
| 'm' | timedelta |
| 'M' | datetime |
| 'O' | (Python) objects |
| 'S', 'a' | (byte-)string |
| 'U' | Unicode |
| 'V' | raw data (void) |
In [63]:
dt = np.dtype('i1')
print(dt)
dt = np.dtype('i2')
print(dt)
dt = np.dtype('i4')
print(dt)
dt = np.dtype('i8')
print(dt)
In [65]:
dt = np.dtype([('we will be using this user later in our examples.','a128')])
print(dt)
In [67]:
user=np.dtype([('first_name','S30'), ('last_name','S30'), ('pay', 'i2'), ('designation_id', 'f2')])
print(user)
In [ ]:
In [82]:
a4 = np.array([[1, 3, 2], [1, 4, 5]], dtype=np.int8)
print(a4)
In [87]:
a4 = np.array([[1, 3, 2], [1, 4, 5]], dtype=np.float)
print(a4)
In [ ]:
# TODO How much memory space a variable is taking
In numpy standard mathematical operations when used with arrays, operate at element level, which makes it mandetory that the arrays should be the same size.
In [95]:
import numpy as np
x = np.array([[1,2],[3,4]], dtype=np.float64)
y = np.array([[5,6],[7,8]], dtype=np.float64)
print(x)
print(y)
In [96]:
print(x + y)
print(np.add(x, y))
In [98]:
print(x - y)
print(np.subtract(x, y))
In [100]:
print(x * y)
print(np.multiply(x, y))
In [102]:
print(x / y)
print(np.divide(x, y))
In [103]:
print(np.sqrt(x))
In [ ]:
In [105]:
am = np.array([[3,5,7,9], [2, 4, 5, 6], [12, 14, 15, 6]], float)
print(am)
print(am.shape)
am1 = np.array([[3,5,7,9], [2, 4, 5, 6]], float)
print(am1)
print(am1.shape)
In the above examples am and am1 are of different shapes, thus any maths operation performed will result in failure as shown in the below maths operations.
In [107]:
am + am1
In [108]:
am - am1
In [109]:
am * am1
Now, lets update the arrays with similar shape and observe the standard maths operations.
In [111]:
am = np.array([[2, 4, 5, 6], [12, 14, 15, 6]], float)
print(am)
print(am.shape)
am1 = np.array([[3,5,7,9], [2, 4, 5, 6]], float)
print(am1)
print(am1.shape)
In [112]:
am + am1
Out[112]:
In [113]:
am - am1
Out[113]:
In [114]:
am * am1
Out[114]:
In [115]:
am ** am1
Out[115]:
In [116]:
am % am1
Out[116]:
In [117]:
am / am1
Out[117]:
In [121]:
a = np.array([[1, 2 , 3], [3, 4, 3], [5, 6, 5]], float)
b = np.array([-1, 3, 3], float)
print(a.shape)
print(b.shape)
In [119]:
a + b
Out[119]:
but, following example will fail.
In [124]:
a = np.array([[1, 2 , 3], [3, 4, 3], [5, 6, 5]], float)
b = np.array([[-1, 3, 3], [-1, 3, 3]], float)
print(a.shape)
print(b.shape)
In [12]:
a+b
in order for the 'a' to work they should be something similar to below example
In [126]:
a = np.array([[[1, 2 , 3], [3, 4, 3]], [[5, 6, 5], [6,7,6]]], float)
b = np.array([[-1, 3, 3], [-1, 3, 3]], float)
print(a.shape)
print(b.shape)
and the below code will work without any issue as smaller array can be repeated on the larger array.
In [128]:
a + b
Out[128]:
In [137]:
a = np.array([[[1, 2 , 3], [3, 4, 3]], [[5, 6, 5], [6,7,6]]], float)
b = np.array([[2, 3, 3], [ 1, 5, 1]], float)
print(a.shape)
print(b.shape)
In [138]:
a + b
Out[138]:
Numpy also offers large library of common mathematical functions that can be applied elementwise to arrays.
Some of the other maths related functions are as follows:
abs, sign, sqrt, exp, (log, log10), (sin, cos, tan, arcsin, arccos, arctan), (sinh, cosh, tanh, arcsinh, arccosh and arctanh).
In [ ]:
np.sqrt(a)
In [14]:
np.sqrt(a + b)
Out[14]:
Maths constants such as pi are also provided
In [15]:
print(np.pi)
print(np.e)
Numpy also provide functions to find the lower, upper, or nearest (rounded) integer through floor, ceil, and rint
In [152]:
a = np.array([1, 4, 5, 66.50, 77.77, 334.22], float)
In [153]:
print(a)
np.rint(a)
Out[153]:
In [150]:
np.floor(a)
Out[150]:
In [154]:
np.ceil(a)
Out[154]:
Just like normal array np array's can also be iterated
In [155]:
for i in a:
print(i, "....")
In [21]:
for multi-dimensional array, multiple assinations is also available
In [162]:
am = np.array([[2, 4, 5, 6], [12, 14, 15, 6]], float)
print(am)
print(am.shape)
In [163]:
for i in am:
print(i, "....")
In [161]:
am2 = np.array([[[2, 4],[ 5, 6]], [[12, 14], [15, 6]]], float)
print(am2)
print(am2.shape)
In [164]:
for i in am2:
print(i, "....")
In [159]:
for i, j , k, l in am:
print(i, j, k, l)
This works; however when the matrix x is very large, computing an explicit loop in Python could be slow. Note that adding the vector v to each row of the matrix x is equivalent to forming a matrix vv by stacking multiple copies of v vertically, then performing elementwise summation of x and vv. We could implement this approach like this:
In [7]:
import numpy as np
# We will add the vector v to each row of the matrix x,
# storing the result in the matrix y
x = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])
v = np.array([1, 4, 2])
vv = np.tile(v, (4, 1)) # Stack 4 copies of v on top of each other
print(x)
print(vv)
print(vv.shape)
print(x.shape)
y = x + vv # Add x and vv elementwise
print(y)
In [8]:
y1 = x + v
print(y1)
Broadcasting two arrays together follows these rules:
Functions that support broadcasting are known as universal functions. The list of all universal functions is listed at http://docs.scipy.org/doc/numpy/reference/ufuncs.html#available-ufuncs
In [38]:
import numpy as np
# Compute outer product of vectors
v = np.array([1,2,3]) # v has shape (3,)
w = np.array([4,5]) # w has shape (2,)
# To compute an outer product, we first reshape v to be a column
# vector of shape (3, 1); we can then broadcast it against w to yield
# an output of shape (3, 2), which is the outer product of v and w:
# [[ 4 5]
# [ 8 10]
# [12 15]]
print(np.reshape(v, (3, 1)) * w)
# Add a vector to each row of a matrix
x = np.array([[1,2,3], [4,5,6]])
# x has shape (2, 3) and v has shape (3,) so they broadcast to (2, 3),
# giving the following matrix:
# [[2 4 6]
# [5 7 9]]
print(x + v)
# Add a vector to each column of a matrix
# x has shape (2, 3) and w has shape (2,).
# If we transpose x then it has shape (3, 2) and can be broadcast
# against w to yield a result of shape (3, 2); transposing this result
# yields the final result of shape (2, 3) which is the matrix x with
# the vector w added to each column. Gives the following matrix:
# [[ 5 6 7]
# [ 9 10 11]]
print((x.T + w).T)
# Another solution is to reshape w to be a column vector of shape (2, 1);
# we can then broadcast it directly against x to produce the same
# output.
print(x + np.reshape(w, (2, 1)))
# Multiply a matrix by a constant:
# x has shape (2, 3). Numpy treats scalars as arrays of shape ();
# these can be broadcast together to shape (2, 3), producing the
# following array:
# [[ 2 4 6]
# [ 8 10 12]]
print(x * 2)
In [ ]:
In [ ]:
In [ ]:
In [ ]:
Apart from computing mathematical functions using arrays, we frequently need to reshape or otherwise manipulate data in arrays. The simplest example of this type of operation is transposing a matrix; to transpose a matrix, simply use the T attribute of an array object:
In [18]:
x = np.array([[1,2], [3,4]])
print(x)
print(x.T)
# Note that taking the transpose of a rank 1 array does nothing:
v = np.array([1,2,3])
print(v)
print(v.T)
Broadcasting is a powerful mechanism that allows numpy to work with arrays of different shapes when performing arithmetic operations. Frequently we have a smaller array and a larger array, and we want to use the smaller array multiple times to perform some operation on the larger array.
For example, suppose that we want to add a constant vector to each row of a matrix. We could do it like this:
In [9]:
x = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])
v = np.array([1, 0, 1])
y = np.empty_like(x) # Create an empty matrix with the same shape as x
# Add the vector v to each row of the matrix x with an explicit loop
for i in range(4):
y[i, :] = x[i, :] + v
# Now y is the following
# [[ 2 2 4]
# [ 5 5 7]
# [ 8 8 10]
# [11 11 13]]
print(y)
In [12]:
y2 = x + v
print(x)
print(v)
print(y2)
NumPy makes it possible to test to see if rows match certain values using mathematical comparison operations like <, >, >=, <=, and ==.
In [4]:
print(a)
In [2]:
a > 67
In [3]:
a = np.array([1, 3, 5, 7])
b = np.array([1, 3, 5, 7])
c = np.array([1, 3, 5, 2])
In [6]:
print(a == b)
print(c == b)
print(c == a)
In [10]:
print(a > b)
print(c > b)
print(c > a)
In [11]:
print(a < b)
print(c < b)
print(c < a)
In [8]:
print(a >= b)
print(c >= b)
print(c >= a)
In [9]:
print(a <= b)
print(c <= b)
print(c <= a)
In [ ]:
One of the powerful things we can do with a Boolean array and a NumPy array is select only certain rows or columns in the NumPy array
In [4]:
a_sub = a > 67
print(a_sub)
In [5]:
a[a_sub]
Out[5]:
We can change the shape of arrays while still preserving all of their elements. This often can make it easier to access array elements. The simplest reshaping is to flip the axes, so rows become columns, and vice versa. We can accomplish this with the numpy.transpose function:
In [167]:
b = np.array([[1,2], [3, 4], [5, 6]])
np.transpose(b).shape
Out[167]:
We can use the numpy.ravel function to turn an array into a one-dimensional representation. It will essentially flatten an array into a long sequence of values:
In [16]:
b.ravel()
Out[16]:
the numpy.reshape function to reshape an array to a certain shape we specify. The below code will turn the second row of wines into a 2-dimensional array with 2 rows & 3 cols
In [174]:
c = b.reshape((3,2))
print(c)
print(c.shape)
In [12]:
c = b.reshape((4,2))
print(c)
print(c.shape)
In [18]:
a = np.triu(np.ones((4,4)), 1)
print(a)
In [17]:
print(a.T)
In [20]:
a = np.array(range(10))
print(a)
In [22]:
print(a.sum())
Now lets try to do get the sum for n-dimension array
sum by rows and by columns
In [15]:
x = np.array([[2, 4], [6, 8]])
print(x)
print(x.sum())
In [16]:
print(x.sum(axis=0))
In [17]:
print(x.sum(axis=1))
In [18]:
x[0, :].sum(), x[1,:].sum()
Out[18]:
In [46]:
x[:, 0].sum(), x[:, 1].sum()
Out[46]:
In [38]:
Out[38]:
In [ ]:
In [20]:
x = np.array([[2, 4], [6, 8], [1 , 5]])
print(x)
In [21]:
print(x.sum(axis=0))
In [22]:
print(x.sum(axis=1))
In [23]:
y = np.array([[[2, 4], [6, 8]], [[1 , 5], [1 , 15]]])
print(y)
In [24]:
print(y.sum(axis=0))
In [6]:
print(y.sum(axis=1))
In [59]:
x = np.random.rand(2, 2, 3)*10
print(x)
In [60]:
print(x.sum())
print(x.sum(axis=1))
In [66]:
print(x[1, :].sum())
print(x[0, :].sum())
We can also use the following method to get the same result
In [67]:
np.array([x[:, 0].sum(), x[:, 1].sum()]).sum()
Out[67]:
In [65]:
np.array([x[0, :].sum(), x[1, :].sum()]).sum()
Out[65]:
In [71]:
# TODO : more examples on multi-D
In [2]:
import numpy as np
b = np.random.rand(3, 1, 2)*10
print(b)
In [3]:
b.min()
Out[3]:
In [4]:
b.max()
Out[4]:
In [5]:
b.argmin() # index of minimum
Out[5]:
In [7]:
b.argmax() # index of max
Out[7]:
In [25]:
np.all([True, True, False])
Out[25]:
In [27]:
np.all([False, False, False])
Out[27]:
In [28]:
np.all([True, True, True])
Out[28]:
In [29]:
np.any([True, True, False])
Out[29]:
In [30]:
np.any([False, False, False])
Out[30]:
In [31]:
a = np.zeros((100, 100))
print(a)
In [32]:
print(np.any(a != 0))
print(np.any(a == 0))
In [33]:
np.all(a == a)
Out[33]:
Note: Can be used for array comparisons:
In [38]:
a = np.array([1, 2, 3, 2])
b = np.array([2, 2, 3, 2])
c = np.array([6, 4, 4, 5])
((a <= b) & (b <= c)).all()
Out[38]:
In [39]:
((a >= b) & (b <= c)).all()
Out[39]:
In [37]:
((a >= b) & (b <= c)).any()
Out[37]:
In [40]:
x = np.array(np.random.rand(2, 2, 3)*10)
y = np.array(np.random.rand(2, 2, 3)*10)
print(x)
print(y)
print(x.mean())
print(y.mean())
In [41]:
print(np.median(x))
print(np.median(y))
In [30]:
print(np.median(y, axis=-1))
In [32]:
x.std() # full population standard dev.
Out[32]:
In [ ]:
Return the cumulative sum of the elements along a given axis. It returns A new array holding the result is returned unless out is specified, in which case a reference to out is returned. The result has the same size as a, and the same shape as a if axis is not None or a is a 1-d array.
In [43]:
a = np.cumsum(x)
print(a)
In [36]:
np.cumsum(x,axis=1)
Out[36]:
In [37]:
np.cumsum(a,axis=0)
Out[37]:
In [ ]:
In [1]:
!cat code/populations.txt
In [4]:
data = np.loadtxt('code/populations.txt')
In [5]:
year, hares, lynxes, carrots = data.T # trick: columns to variables
In [7]:
from matplotlib import pyplot as plt
%matplotlib inline
plt.axes([0.2, 0.1, 0.5, 0.8])
plt.plot(year, hares, year, lynxes, year, carrots)
plt.legend(('Hare', 'Lynx', 'Carrot'), loc=(1.05, 0.5))
Out[7]:
In [8]:
populations = data[:, 1:]
populations.mean(axis=0)
Out[8]:
In [9]:
populations.std(axis=0)
Out[9]:
In [11]:
#Which species has the highest population each year?:
np.argmax(populations, axis=1)
Out[11]:
In [45]:
a = np.array([[1, 2, 3], [4, 5, 6]])
a.ravel()
Out[45]:
In [46]:
print(a.shape)
b = a.ravel()
print(b)
b = b.reshape((2, 3))
print(b)
print( b.reshape((3,2)))
print( b.reshape((1,6)))
print( b.reshape((6, 1)))
In [24]:
print( b.reshape((2,2))) # should error out
In [ ]:
# Provide the reason for the error
In [18]:
b[0, 0] = 9911
print(a)
In [ ]:
In [28]:
a = np.arange(4*3*2).reshape(4, 3, 2)
print(a)
print(a.shape)
In [29]:
b = a.transpose(1, 2, 0)
print(b)
print(b.shape)
In [31]:
a = np.arange(4)
print(a)
In [33]:
a.resize((8,))
print(a)
In [35]:
### NOTE: However, it must not be referred to somewhere else:
b = a
a.resize((6,))
In [43]:
a = np.array([[4, 3, 5], [1, 2, 1]])
In [ ]:
b = np.sort(a, axis=1)
print(b)
In [38]:
# in-place sorting
# Note: Sorts each row separately!
a.sort(axis=1)
print(a)
In [44]:
a.sort(axis=0)
print(a)
In [6]:
# Sorting with indexing
a = np.array([4, 3, 1, 2])
j = np.argsort(a)
print(j)
print(a[j])
## Finding minima and maxima
print(np.argmin(a), np.argmax(a))
--
In [ ]:
In [ ]:
In [ ]:
In [ ]:
What is the shape & rank of the following arrays
What is the difference between sum and cumsum?
In [ ]: